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Framework for the assessment of uncommissioned third-party 
evidence 


How the FSA uses evidence 

The FSA prides itself in the transparent use of science and evidence to inform its 
advice and recommendations. Evidence enters the FSA from a variety of sources, 
and through all the Department’s different functions. For example, the FSA conducts 
and funds its own research to keep abreast of changes in the food system and 
identify emerging issues; whilst industry and members of the public can submit 
evidence and views during public consultations. When new advice on food safety is 
required for Government, business or consumers, the Risk Analysis Process will be 
used to assess that risk and advise on its handling. 


Our use and interpretation of scientific evidence and analysis is informed by the 
input, scrutiny and challenge of independent experts, for instance through the FSA’s 


Scientific Advisory Committees (SACs). 


Uncommissioned evidence 

Sometimes evidence may be sent to the FSA by a member of the public, industry 
representative or consumer group outside of the usual research and consultation 
processes. Such uncommissioned evidence might be sent with a variety of aims 
such as filling a perceived gap in knowledge or suggesting a change relevant to a 
policy or legislation. 


When the FSA receives such evidence, it will: 


e be transparent about how such evidence is assessed and used to develop its 
evidence base, policy making and risk communication. 

e assess evidence in its proper context using the principles of quality, trust and 
robustness. 

e seek to minimise bias in its assessments of evidence by using professional 
protocols, its SACs, peer review and/or multi-disciplinary teams 

e be open and transparent about the conclusions it has reached about any 
evidence submitted to it. 


Guidelines 

These guidelines outline the FSA’s expectations concerning the standard of 
uncommissioned evidence that it receives, and provide guidance on how the 
strengths and weaknesses of this evidence will be assessed. They should therefore 
be used as a guide for anyone submitting evidence to the FSA — both those directly 
performing studies, and those choosing existing evidence to support their position. 


The guidelines are not exhaustive as the work that the FSA undertakes covers a 
broad range of disciplines and areas of interest. Links to some of the organisations 
that provide detailed guidance for specific areas and disciplines is provided in the 
Helpful Links section at the end of this document. 


The decisions that the FSA make on food safety are based upon a broad body of 
evidence. When the FSA receives new evidence on an existing issue in food safety it 
will consider it in the context of the body of evidence that has already been used to 
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inform a policy or decision. This will inform the assessment of whether any changes 
to the existing position may be needed. 


In new, emerging or rapidly developing areas, a decision may need to be taken 
based upon limited evidence. The FSA will use the best available evidence to make 
this decision, recognise where there are gaps or limitations in knowledge, and be 
open to change as new evidence becomes available. 


The guidelines are organised by the principles of quality, trust and robustness, as 
outlined below. 


Quality 

Evidence should be reliable and relevant to the question at hand. Clearly defining the 
context of the original study and the question originally asked can help to identify if 
the evidence is relevant. Using well-recognised methods and data analysis can help 
to ensure it is both relevant and reliable. If a novel method is used, a clear 
explanation of why it has been used and what advantage it brings is important. Data 
and analysis should be clearly presented, with a narrative that directly links them to 
the conclusions within the study. 


Clarity 

e All evidence sent to the FSA should be clearly laid out, outlining the study 
approach, the data collected, and analysis performed. 

e |f evidence has been collated from several sources this should be clearly 
indicated, and the method used for its collation and integration described. 

e Precise language should be used to describe the aims of the study or 
research question relating this to the study design and conclusions. 

e Methods should be described in enough detail that they could be 
independently reproduced — including the controls, reference standards and 
quality assurance measures used. This includes both study methods and 
methods for data analysis. 

e Aclear statement should be provided describing how data' were cleaned?, 
processed and analysed, and why such approaches were taken. 

e The conclusions of a study must be based on the evidence presented, with a 
clear narrative linking the data and analysis to those conclusions. 


Relevance 

e To assess the relevance of a study to a particular issue, the FSA will look at 
the context of the original study and the question(s) it was designed to 
answer. As key information about the way the study was conducted will be 
used to assess this, the clarity and transparency of the evidence are therefore 
important. 

e The study design and the methods used should be justified with reference to 
the original question or hypothesis — including how potentially confounding 
variables were controlled for. 


1 We define data as all direct outputs from a study, including both quantitative and qualitative results, 
and digital images used to support analysis and conclusions. 

2 Where data cleaning refers to the detection and removal of incorrect or corrupt data points, 
duplicates or empty fields, and ensuring consistency of units and formatting. 
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Consider the relevance of the study population, specimen or substance to the 
target population, specimen or substance. This is particularly important when 
considering the biological relevance of a study and its conclusions.’ 

If the study is qualitative, a comprehensive description of the context of the 
work should be included. For example, the culture, livelinood, community, 
socio-economic status and environment of participants. 

Statistical analysis is essential in scientific studies. Studies should include a 
clear outline of the methods used, and why they were chosen, with an 
explanation of what question the analysis aimed to answer. Statistical point 
estimates and confidence intervals are recommended alongside significance 
testing. 

Where the evidence relates to a new method, outline the context in which the 
method should be used and why. Where relevant, make clear the advantages 
and drawbacks relative to more established methods. 


Reliability 


Where possible, methods recommended by national and international bodies 
such as the FAO, OECD and the Codex Alimentarius Commission, or 
methods widely used in academic literature should be used. 

Good governance should be practiced when performing research. Refer to 
best practice guidelines such as the OECD’s Principles of Good Laboratory 
Practice. 

Whether routine or not, all methods used should be referenced. If a standard 
method has been adapted, the study should state why and describe the 
differences. If a new method is proposed, a description of how it differs from 
the standard method(s), and where possible a comparative study should be 
provided. 

All evidence must include consideration of uncertainty*. Where possible this 
should be quantified using recognised methods. If the uncertainty is 
associated with an expert judgement, state whether it is qualitative or 
quantitative, and how it was discerned. 

Variability must also be considered, and where possible quantified®. Where 
variability has been controlled for in a study, consider if this affects 
generalisability to the target population, specimen, or substance. 

If mathematical models are used, the results of the sensitivity analysis 
performed should be provided, stating which parameters were tested, which 
were not and why. 


3 The work on Biological Relevance and Statistical Significance led by the Committees on Toxicity, 
Carcinogenicity and Mutagenicity will explore this in further detail. 

4 We use the definition of uncertainty provided by the Committee on Toxicity: as the estimated sum of 
the limits in knowledge. We include limitations to apparatus, experimental techniques, models and 
study designs, as well as essential unpredictability. 

5 Variability is defined as the inherent heterogeneity between individuals or groups, or over time or 
space. This may be humans, animals or other specimens. 


Trust 
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Transparency and impartiality are key in providing confidence that evidence is 
trustworthy. Evidence that is shared transparently will include access to all 
underlying data, a clear explanation of the methods used and why, and the limits to 
the evidence. This includes stating uncertainties, variability and assumptions, 
indicating where results differ from comparable investigations and where there is 
dissenting opinion among experts. Any evidence and its assessment is at risk of 
bias, but this can be mitigated by ensuring that sources of bias are recognised, peer 
review is performed and challenge is built into the assessment process. 


Transparency 


Openness and transparency are core principles of the way the FSA works; 
evidence submitted to the FSA should also demonstrate these principles as 
far as reasonably possible. 

In addition to clearly presenting all relevant data and associated analysis, 
access to the raw and omitted data from the study should be provided. If this 
is not possible, state why. 

Known gaps in the evidence should be stated and limitations to models or 
study designs outlined. This includes assumptions on what is or is not 
important for the question being asked, and therefore what has been included 
or excluded from the study or model design. 

Consider alternative hypotheses and make comparisons to the published 
body of research on the area, stating where results differ or where there is 
disagreement in expert opinion. 

Clearly indicate when evidence is compiled from a range of sources. 
Reference all sources and state the method used to compile the evidence, for 
example, using widely accepted guidelines for evidence synthesis such as 
meta-analysis and systematic review procedures. 


Impartiality and bias 


Increased risk of bias reduces the confidence in the outputs of a piece of 
evidence. 

All potential sources of bias should be clearly described, considering each 
stage of the study and any actions taken to mitigate them should be stated. 
The sources of bias and appropriate mitigating actions will be dependent upon 
on the type of study being performed. 

Where data are omitted from a study report, this should be clearly stated, with 
reasoning. This includes both full data sets and individual data points. If 
evidence is from a range of sources, the way in which sources were chosen 
or omitted should be given. 

Where expert judgement is used, state why, how the experts were chosen 
and the initial question that was asked of them. Any underlying data or 
evidence that the judgement is based upon should be provided, and a 
statement of uncertainty should be included with the judgement. 

If the evidence used is not published in a peer-reviewed journal, any critical 
review that has been performed should be described. 

In all instances, sources of funding and conflicts of interest must be stated. 
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Robustness 

For evidence to be robust, a broad body of evidence should be considered from 
several perspectives, with each piece of evidence weighed based on its quality and 
trustworthiness. The body of evidence will be made more robust if the pieces of 
evidence are reproducible using the same and different methods. If an outcome is 
consistently observed when tested using different methods and populations, this 
provides confidence that the outcome itself is robust. 


Consistency 

e The FSA expects that studies submitted to them should be reproducible. To 
demonstrate reproducibility, describe how tests were replicated and the extent 
of any variation in the observed results. 

e The clarity and transparency of a study, as well as the use of standard 
methods, reference standards and quality control methods can help ensure 
that a study can be repeated by other researchers. 

e |f several independent studies are performed repeating the same or similar 
tests and gaining the same or similar outcomes, this will increase confidence 
in the outcome. 

e The robustness of an outcome can be tested by varying parameters within the 
study, and by using different methods to test the same relationship or 
outcome (triangulation). This may be done in a single study, or by comparing 
the outcomes of several studies. 


Adequacy 

e Explain the importance of the evidence with reference to the broader body of 
evidence to which it contributes. Consider whether evidence highlights any 
gaps in the existing body of evidence, and how much it increases the 
understanding of a new or emerging area. 

e Different types of evidence may need to be combined for a comprehensive 
assessment of an issue to be undertaken®. Consider the other types of 
evidence that are required when assessing an issue and explain how your 
evidence relates to them. 

e The adequacy of a piece of evidence will vary depending on the type of study 
and the question being asked. However, criteria such as the magnitude of any 
effect, the power of a study, and its applicability to the target population, 
specimen or substance may be considered. 

e Significance testing is often used to indicate the magnitude of a result, but it is 
not by itself sufficient to indicate that a piece of evidence is strong or will 
translate to an important real-world impact. Consider the relevance of the 
study and the statistical test to the decision or policy that the evidence is 
being used to address’. 


6 For instance, as described in the Committees on Toxicity and Carcinogenicity’s guidelines on the 
synthesis and integration of epidemiological and toxicological evidence. 

7 The work on Biological Relevance and Statistical Significance led by the Committees on Toxicity, 
Carcinogenicity and Mutagenicity will provide further detail of how this is assessed by SACs. 


Helpful links 


Organisation/group/ 
bod 


FSA 


Regulated product 
authorisation 


EQUATOR network 


OECD 
Good Laboratory 
Practice 
Good in vitro method 
practices 


Food and Agriculture 


Organization of the 
United Nations (FAO 


Codex Alimentarius 
Commission 


UK Accreditation 
Service 


International 
Organization for 
Standardization and 
European Committee 
for Standardization 
International 
Programme on 
Chemical Safet 

UK Statistics 


Authority 
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Area 


Science Advisory 
Committee guidance 
for evidence 
assessment 


Guidance for evidence 
submission 


Guidelines and opinion 


Reporting of health 
research 


Guidance on methods 
and governance in 
laboratory practice 


Guidance and 
methods for risk 
analysis 

Guidance and 
standards for risk 
analysis and the 
maintenance of food 
safety standards 
Accreditation body 


Standards for testing in 
food safety 


Guidance for risk 
assessment 


Code of Practice for 
Statistics 


Guidance for 
assessing evidence 


Description 


Guidance from the SACs is published 
on their pages, including guidance on 
how evidence should be assessed and 
integrated 


Guidance on what is required when 
providing evidence for approval, for 
instance for a regulated product 
European Food Safety Authority Journal 
— through which all guidelines and 
opinion are published 

Library of reporting guidelines for the 
most commonly used study types in 
health research. 

Principles and technical guidance for 
laboratory practice used for regulatory 
practice, including the Mutual 
Acceptance of Data. 


Technical guidance for methods to be 
used during risk assessment for food 
safety. 

Codex standards include guidelines and 
recommendations for assessing and 
implementing food safety. 


Accredits organisations providing 
laboratory testing services for regulatory 
purposes, including the methods used. 
ISO and CEN develop international 
standards for testing in food safety. 


A toolkit for the assessment of risk from 
exposure to chemicals 


Code of practice for the production and 
release of official statistics, that can be 
voluntarily applied by any organisation 
Detailed guidelines for weighing 
evidence in a health and social care 
context 





